預設 EKS cluster 使用 Amazon VPC Container Network Interface(CNI)Plugin 作為 CNI。EKS 集群建置後,我們即可以透過 kubectl 命令查看 DaemonSet aws-node
而無需手動安裝 CNI plugin。
$ kubectl -n kube-system get ds
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
aws-node 3 3 3 3 3 <none> 5d11h
kube-proxy 3 3 3 3 3 <none> 5d11h
本文將探討「為什麼 EKS cluster 知道預設 CNI Plugin 為 Amazon VPC CNI plugin」,希望理解 EKS CNI plugin 設定過程。
與上一篇查看 kubelet
systemd unit 設定檔時,我們其實已經看過 kubelet 啟用 --network-plugin=cni
。kubelet 將會讀取 --network-plugin-dir
目錄設定檔,並使用 CNI 設定檔設置每一個 Pod network。
[ec2-user@ip-192-168-65-212 ~]$ systemctl cat kubelet
# /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=docker.service iptables-restore.service
Requires=docker.service
[Service]
ExecStartPre=/sbin/iptables -P FORWARD ACCEPT -w 5
ExecStart=/usr/bin/kubelet --cloud-provider aws \
--config /etc/kubernetes/kubelet/kubelet-config.json \
--kubeconfig /var/lib/kubelet/kubeconfig \
--container-runtime docker \
--network-plugin cni $KUBELET_ARGS $KUBELET_EXTRA_ARGS
...
...
...
--cni-bin-dir="/opt/cni/bin"
:kubelet 啟用時會查看此目錄--cni-conf-dir="/etc/cni/net.d"
[ec2-user@ip-192-168-65-212 ~]$ journalctl -u kubelet | grep -e "--cni"
Sep 19 09:46:56 ip-192-168-65-212.eu-west-1.compute.internal kubelet[3347]: I0919 09:46:56.765003 3347 flags.go:59] FLAG: --cni-bin-dir="/opt/cni/bin"
Sep 19 09:46:56 ip-192-168-65-212.eu-west-1.compute.internal kubelet[3347]: I0919 09:46:56.765008 3347 flags.go:59] FLAG: --cni-cache-dir="/var/lib/cni/cache"
Sep 19 09:46:56 ip-192-168-65-212.eu-west-1.compute.internal kubelet[3347]: I0919 09:46:56.765014 3347 flags.go:59] FLAG: --cni-conf-dir="/etc/cni/net.d"
不免俗的來驗證一下,我們可以直接透過 EC2 啟用 Amazon EKS optimized Amazon Linux AMI 來比對:
amazon/amazon-eks-node-1.22-v20220824
[ec2-user@ip-172-31-44-117 ~]$ ls /opt/cni/bin
bandwidth bridge dhcp firewall flannel host-device host-local ipvlan loopback macvlan portmap ptp sbr static tuning vlan
[ec2-user@ip-172-31-44-117 ~]$ ls -al /etc/cni/net.d/
ls: cannot access /etc/cni/net.d/: No such file or directory
[ec2-user@ip-192-168-65-212 ~]$ ls /opt/cni/bin
aws-cni aws-cni-support.sh bandwidth bridge dhcp egress-v4-cni firewall flannel host-device host-local ipvlan loopback macvlan portmap ptp sbr static tuning vlan
[ec2-user@ip-192-168-65-212 ~]$ sudo cat /etc/cni/net.d/10-aws.conflist
{
"cniVersion": "0.3.1",
"name": "aws-cni",
"plugins": [
{
"name": "aws-cni",
"type": "aws-cni",
"vethPrefix": "eni",
"mtu": "9001",
"pluginLogFile": "/var/log/aws-routed-eni/plugin.log",
"pluginLogLevel": "DEBUG"
},
{
"name": "egress-v4-cni",
"type": "egress-v4-cni",
"mtu": 9001,
"enabled": "false",
"nodeIP": "192.168.65.212",
"ipam": {
"type": "host-local",
"ranges": [[{"subnet": "169.254.172.0/22"}]],
"routes": [{"dst": "0.0.0.0/0"}],
"dataDir": "/run/cni/v6pd/egress-v4-ipam"
},
"pluginLogFile": "/var/log/aws-routed-eni/egress-v4-plugin.log",
"pluginLogLevel": "DEBUG"
},
{
"type": "portmap",
"capabilities": {"portMappings": true},
"snat": true
}
]
}
/opt/cni/bin
路徑正如同 CNI 參數名稱為 CNI binary 路徑。預設 EKS 在製作 worker node AMI 時,透過 script [3] 安裝 常見 CNI plugin binary [4]。上述比對可以了解,在 aws-node
Pod 尚未執行之前,aws-cni
binary 及設定檔 /etc/cni/net.d/10-aws.conflist
皆尚未被安裝或建立。
接續,我們也可以透過 kubectl 檢視 aws-node
Pod logs,可以看到由 Amazon VPC CNI plugin 預設使用的 initContainers amazon-k8s-cni-init
安裝 loopback
、portmap
、bandwidth
及 aws-cni-support.sh
plugin。init container 執行結束後,aws-node
也有執行複製設定檔及 binary。以下為 logs:
$ kubectl -n kube-system logs aws-node-5c6w5 --all-containers --timestamps
2022-09-19T09:48:57.333672597Z Copying CNI plugin binaries ...
2022-09-19T09:48:57.333810096Z + PLUGIN_BINS='loopback portmap bandwidth aws-cni-support.sh'
2022-09-19T09:48:57.333831182Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.333835571Z + '[' '!' -f loopback ']'
2022-09-19T09:48:57.333838091Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.333840648Z + '[' '!' -f portmap ']'
2022-09-19T09:48:57.333843085Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.333845490Z + '[' '!' -f bandwidth ']'
2022-09-19T09:48:57.333847874Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.333850503Z + '[' '!' -f aws-cni-support.sh ']'
2022-09-19T09:48:57.333908878Z + HOST_CNI_BIN_PATH=/host/opt/cni/bin
2022-09-19T09:48:57.333911360Z + echo 'Copying CNI plugin binaries ... '
2022-09-19T09:48:57.333914044Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.333916695Z + install loopback /host/opt/cni/bin
2022-09-19T09:48:57.358632542Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.358667213Z + install portmap /host/opt/cni/bin
2022-09-19T09:48:57.371180218Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.371397188Z + install bandwidth /host/opt/cni/bin
2022-09-19T09:48:57.386902686Z + for b in '$PLUGIN_BINS'
2022-09-19T09:48:57.387089749Z + install aws-cni-support.sh /host/opt/cni/bin
2022-09-19T09:48:57.389252423Z + echo 'Configure rp_filter loose... '
2022-09-19T09:48:57.389584940Z Configure rp_filter loose...
2022-09-19T09:48:57.389945375Z ++ get_metadata local-ipv4
2022-09-19T09:48:57.391264947Z +++ curl -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60'
2022-09-19T09:48:57.473320665Z % Total % Received % Xferd Average Speed Time Time Time Current
2022-09-19T09:48:57.473705220Z Dload Upload Total Spent Left Speed
100 56 100 56 0 0 28000 0 --:--:-- --:--:-- --:--:-- 56000
2022-09-19T09:48:57.478526895Z ++ TOKEN=AQAEAJKJMlzdfIbRn8BQh4U5I2i0dMhKrW4DxRx64XuuP_g4rPzuOw==
2022-09-19T09:48:57.478907025Z ++ attempts=60
2022-09-19T09:48:57.478914505Z ++ false
2022-09-19T09:48:57.478917627Z ++ '[' 1 -gt 0 ']'
2022-09-19T09:48:57.478920837Z ++ '[' 60 -eq 0 ']'
2022-09-19T09:48:57.479064447Z +++ curl -H 'X-aws-ec2-metadata-token: AQAEAJKJMlzdfIbRn8BQh4U5I2i0dMhKrW4DxRx64XuuP_g4rPzuOw==' http://169.254.169.254/latest/meta-data/local-ipv4
2022-09-19T09:48:57.486678298Z % Total % Received % Xferd Average Speed Time Time Time Current
2022-09-19T09:48:57.486692175Z Dload Upload Total Spent Left Speed
100 14 100 14 0 0 14000 0 --:--:-- --:--:-- --:--:-- 14000
2022-09-19T09:48:57.488090438Z ++ meta=192.168.65.212
2022-09-19T09:48:57.488210992Z ++ '[' 0 -gt 0 ']'
2022-09-19T09:48:57.488331224Z ++ '[' 0 -gt 0 ']'
2022-09-19T09:48:57.488430469Z ++ echo 192.168.65.212
2022-09-19T09:48:57.488788529Z + HOST_IP=192.168.65.212
2022-09-19T09:48:57.489299361Z ++ get_metadata mac
2022-09-19T09:48:57.489716539Z +++ curl -X PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 60'
2022-09-19T09:48:57.497186993Z % Total % Received % Xferd Average Speed Time Time Time Current
2022-09-19T09:48:57.497200745Z Dload Upload Total Spent Left Speed
100 56 100 56 0 0 56000 0 --:--:-- --:--:-- --:--:-- 56000
2022-09-19T09:48:57.499248303Z ++ TOKEN=AQAEAJKJMlxeFjZlUL_0INYoCXkf7UWmVm4nIKV6nnDeG_VvdZ9-Ig==
2022-09-19T09:48:57.499358341Z ++ attempts=60
2022-09-19T09:48:57.499440488Z ++ false
2022-09-19T09:48:57.499594430Z ++ '[' 1 -gt 0 ']'
2022-09-19T09:48:57.499599969Z ++ '[' 60 -eq 0 ']'
2022-09-19T09:48:57.500015110Z +++ curl -H 'X-aws-ec2-metadata-token: AQAEAJKJMlxeFjZlUL_0INYoCXkf7UWmVm4nIKV6nnDeG_VvdZ9-Ig==' http://169.254.169.254/latest/meta-data/mac
2022-09-19T09:48:57.511676682Z % Total % Received % Xferd Average Speed Time Time Time Current
2022-09-19T09:48:57.511709736Z Dload Upload Total Spent Left Speed
100 17 100 17 0 0 17000 0 --:--:-- --:--:-- --:--:-- 17000
2022-09-19T09:48:57.515247164Z ++ meta=06:1b:3e:40:af:fd
2022-09-19T09:48:57.515648955Z ++ '[' 0 -gt 0 ']'
2022-09-19T09:48:57.515656468Z ++ '[' 0 -gt 0 ']'
2022-09-19T09:48:57.515659528Z ++ echo 06:1b:3e:40:af:fd
2022-09-19T09:48:57.515827055Z + PRIMARY_MAC=06:1b:3e:40:af:fd
2022-09-19T09:48:57.516696407Z ++ grep -F 'link/ether 06:1b:3e:40:af:fd'
2022-09-19T09:48:57.516908826Z ++ awk '-F[ :]+' '{print $2}'
2022-09-19T09:48:57.517104498Z ++ ip -o link show
2022-09-19T09:48:57.529715651Z + PRIMARY_IF=eth0
2022-09-19T09:48:57.529881853Z + sysctl -w net.ipv4.conf.eth0.rp_filter=2
2022-09-19T09:48:57.549511061Z net.ipv4.conf.eth0.rp_filter = 2
2022-09-19T09:48:57.550035252Z + cat /proc/sys/net/ipv4/conf/eth0/rp_filter
2022-09-19T09:48:57.552013538Z 2
2022-09-19T09:48:57.552296409Z + '[' false == true ']'
2022-09-19T09:48:57.552638323Z + sysctl -e -w net.ipv4.tcp_early_demux=1
2022-09-19T09:48:57.557945154Z net.ipv4.tcp_early_demux = 1
2022-09-19T09:48:57.559523168Z + '[' false == true ']'
2022-09-19T09:48:57.560537081Z + echo 'CNI init container done'
2022-09-19T09:48:57.560663349Z CNI init container done
2022-09-19T09:48:58.240326675Z {"level":"info","ts":"2022-09-19T09:48:58.237Z","caller":"entrypoint.sh","msg":"Validating env variables ..."}
2022-09-19T09:48:58.241282124Z {"level":"info","ts":"2022-09-19T09:48:58.240Z","caller":"entrypoint.sh","msg":"Install CNI binaries.."}
2022-09-19T09:48:58.316896574Z {"level":"info","ts":"2022-09-19T09:48:58.316Z","caller":"entrypoint.sh","msg":"Starting IPAM daemon in the background ... "}
2022-09-19T09:48:58.318121492Z {"level":"info","ts":"2022-09-19T09:48:58.317Z","caller":"entrypoint.sh","msg":"Checking for IPAM connectivity ... "}
2022-09-19T09:49:00.401119692Z {"level":"info","ts":"2022-09-19T09:49:00.397Z","caller":"entrypoint.sh","msg":"Retrying waiting for IPAM-D"}
2022-09-19T09:49:00.441859850Z {"level":"info","ts":"2022-09-19T09:49:00.441Z","caller":"entrypoint.sh","msg":"Copying config file ... "}
2022-09-19T09:49:00.450179414Z {"level":"info","ts":"2022-09-19T09:49:00.449Z","caller":"entrypoint.sh","msg":"Successfully copied CNI plugin binary and config file."}
2022-09-19T09:49:00.451045960Z {"level":"info","ts":"2022-09-19T09:49:00.450Z","caller":"entrypoint.sh","msg":"Foregrounding IPAM daemon ..."}
在 Amazon VPC CNI plugin entrypoint [5],可以確認其 aws-cni
binary 為自身啟用時會複製至本地端,同時也能查看到 volumeMount mount 相應目錄。
$ kubectl -n kube-system get ds aws-node -o yaml
...
...
image: 602401143452.dkr.ecr.eu-west-1.amazonaws.com/amazon-k8s-cni:v1.10.1-eksbuild.1
...
...
volumeMounts:
- mountPath: /host/opt/cni/bin
name: cni-bin-dir
- mountPath: /host/etc/cni/net.d
name: cni-net-dir
- mountPath: /host/var/log/aws-routed-eni
name: log-dir
- mountPath: /var/run/aws-node
name: run-dir
- mountPath: /var/run/dockershim.sock
name: dockershim
- mountPath: /run/xtables.lock
name: xtables-lock
...
...
volumes:
- hostPath:
path: /opt/cni/bin
type: ""
name: cni-bin-dir
- hostPath:
path: /etc/cni/net.d
type: ""
name: cni-net-dir
- hostPath:
path: /var/run/dockershim.sock
type: ""
name: dockershim
- hostPath:
path: /run/xtables.lock
type: ""
name: xtables-lock
- hostPath:
path: /var/log/aws-routed-eni
type: DirectoryOrCreate
name: log-dir
- hostPath:
path: /var/run/aws-node
type: DirectoryOrCreate
name: run-dir
VPC CNI plugin 主要有兩個元件:
ipamd
:長期執行於節點本地端的 IP Address Management (IPAM) daemon 主要負責以下兩件事:
在 Amazon VPC CNI plugin entrypoint [5] 的註解裡解釋:一般來說,kubelet 在查看 well-known directory(這邊指的是 /opt/cni/bin
及 /etc/cni/net.d
)時,就會將 CNI plugin 視為 Ready 狀態。然而因為 VPC CNI plugin 提供了上述兩個元件,希望在成功啟動 IPAM daemon 可以確保連接上 Kubernetes 及本地端 EC2 metadata service,並將 aws-cni
binary 複製到 well-known directory。
透過 CloudWatch Logs insight syntax 檢視 kube-apiserver-audit
logs,並且過濾 verb 為 create。
filter @logStream like /^kube-apiserver-audit/
| fields @timestamp, @message
| sort @timestamp asc
| filter objectRef.name == 'aws-node' AND objectRef.resource == 'daemonsets' AND verb == 'create'
| limit 10000
能發現 username 為 eks: cluster-bootstrap
透過 kubectl
方式部署 create 了 aws-node
DaemonSet。
userAgent
: kubectl/v1.22.12 (linux/amd64) kubernetes/dade57b
user.username
: eks: cluster-bootstrap
由上述驗證後,我們可以了解 EKS 在 cluster bootstrap 過程中透過 kubectl 方式部署 VPC CNI plugin。而在 worker node 層級,設定了 kubelet CNI 相關參數,並由 VPC CNI plugin 複製 binary 檔案至主機上使用。
上述資訊透過 EKS 所提供 Logs 來驗證上游 Kubernetes 運作原理,倘若上述內文有所錯誤,隨時可以留言或是私訊我。